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Nonresponse  follow-up — where  enumerators  from  the  Bureau  of  the 
Census  went  door-to-door  to  count  those  individuals  who  did  not  mail  back 
their  questioimaires — was  the  most  costly  and  labor  intensive  of  all  2000 
Census  operations.  According  to  bureau  data,  labor,  mileage,  and  certain 
administrative  costs  alone  amounted  to  about  $1.4  billion,  or  about  22 
percent  of  the  total  $6.5  billion  allocated  for  the  2000  Census  from  fiscal 
year  1991  through  fiscal  year  2003.  In  terms  of  employment,  the  bureau 
hired  about  a  half  a  million  enumerators,  which  temporarily  made  it  one  of 
the  nation’s  largest  employers,  surpassed  by  only  a  handful  of  big 
organizations  like  Wal-Mart  and  the  U.S.  Postal  Service.  Moreover,  the 
workload  and  schedule  of  nonresponse  follow-up — the  need  to  collect  data 
from  about  42  million  nonresponding  households  within  a  10-week  time 
frame — made  the  conduct  of  this  operation  extraordinarily  difficult  and 
complex. 

In  our  prior  work  we  noted  that  the  success  of  nonresponse  follow-up 
would  depend  in  large  part  on  the  bureau’s  ability  to  maintain  data  quality 
while  completing  the  operation  on  schedule,  before  error  rates  increased 
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as  people  moved  or  had  trouble  recalling  who  was  living  at  their  homes  on 
Census  Day — ^April  1.  Timeliness  was  also  important  for  keeping 
subsequent  census  operations  on-track.  In  particular,  this  included  the 
Accuracy  and  Coverage  Evaluation  (A.C.E.),  which  was  a  separate  sample 
survey  designed  to  assess  the  quality  of  the  population  data  collected  in  the 
2000  Census.  For  methodological  reasons,  the  bureau  needed  to  complete 
its  field  data  collection  workload  for  nonresponse  follow-up  before  A.C.E. 
field  data  collection  could  begin. 

To  its  credit,  the  bureau  generally  completed  nonresponse  follow-up 
consistent  with  its  operational  plan.  Nationwide,  according  to  bureau  data, 
the  511  local  census  offices  located  in  the  50  states  generally  completed 
nonresponse  follow-up  in  slightly  less  time  than  the  bureau’s  planned  10- 
week  schedule.  This  was  a  noteworthy  accomplishment  given  the 
operational  uncertainties  the  bureau  faced,  and  stands  in  sharp  contrast  to 
the  bureau’s  1990  experience  when  nonresponse  follow-up  was  hampered 
by  unanticipated  workload  and  staffing  problems  and  was  completed  6 
weeks  behind  schedule. 

This  report  is  the  latest  in  our  series  of  reviews  that  examine  the  results  of 
key  census-taking  operations  and  highlight  opportunities  for  reform  (see 
the  last  page  of  this  report  for  a  list  of  products  issued  to  date).  Our 
objectives  were  to  identify  (1)  practices  that  contributed  to  the  timely 
completion  of  nonresponse  follow-up  and  (2)  lessons  learned  in 
implementing  these  practices  that  the  bureau  may  want  to  consider  as  it 
plans  for  nonresponse  follow-up  during  the  next  census  in  2010. 


Results  in  Brief  several  practices  were  critical  to  the  bureau’s  timely  completion  of 

nonresponse  follow-up.  The  bureau 

•  had  an  aggressive  outreach  and  promotion  campaign,  simplified 
questionnaire,  and  other  efforts  to  boost  the  mail  response  rate  and  thus 
reduce  the  bureau’s  nonresponse  follow-up  workload; 

•  used  a  flexible  human  capital  strategy  that  enabled  it  to  meet  its  national 
recruiting  and  hiring  goals  and  position  enumerators  where  they  were 
most  needed; 

•  called  on  local  census  offices  to  identify  local  enumeration  challenges, 
such  as  locked  apartment  buildings  and  gated  communities,  and  to 
develop  action  plans  to  address  them;  and 

•  applied  ambitious  interim  “stretch”  goals  that  encouraged  local  census 
offices  to  finish  80  percent  of  their  nonresponse  follow-up  workload 


Page  2 


GAO-02-196  2000  Census  Best  Practices  and  Lessons  Learned 


within  the  first  4  weeks  and  be  completely  finished  by  the  end  of  the  8th 
week,  as  opposed  to  the  10-week  time  frame  specified  in  the  bureau’s 
master  schedule. 


Although  these  initiatives  were  key  to  meeting  nonresponse  follow-up’s 
tight  time  frames,  the  bureau’s  experience  in  implementing  them  highlights 
several  significant  challenges  that  lie  ahead  for  the  next  census  in  2010. 
First,  maintaining  the  response  rate  is  becoming  increasingly  expensive. 
While  the  bureau  achieved  similar  response  rates  in  1990  and  2000  (65 
percent  in  1990  and  64  percent  in  2000),  the  bureau  spent  far  more  money 
on  outreach  and  promotion  in  2000:  about  $3.19  per  household  in  2000 
compared  to  $0.88  in  1990  (in  constant  fiscal  year  2000  dollars).  Moreover, 
given  a  variety  of  social,  demographic,  and  attitudinal  trends,  such  as 
changes  in  household  makeup  and  stability,  concerns  over  privacy,  and  an 
increasing  non-English-speaking  population,  achieving  comparable  results 
in  2010  will  likely  require  an  even  larger  investment  of  bureau  resources. 

Second,  public  participation  in  the  census  remains  problematic.  Indeed, 
preliminary  data  on  the  mail  return  rate — a  more  precise  indicator  of 
public  cooperation  with  the  census  than  the  mail  response  rate — declined 
from  74  percent  to  72  percent  from  1990  to  2000.^  Also,  there  still  appears 
to  be  a  large  gap  between  the  relatively  large  number  of  people  who  were 
aware  of  the  2000  Census  and  those  that  actually  responded.  Bridging  this 
gap  has  been  a  longstanding  difficulty  for  the  bureau. 

Third,  the  address  lists  used  for  nonresponse  follow-up  did  not  always 
contain  the  latest  available  information,  in  part  because  the  bureau  found  it 
was  infeasible  to  remove  many  late-responding  households.  As  a  result, 
enumerators  needed  to  visit  over  773,000  households  that  had  already 
mailed  back  their  questionnaires — an  effort  that  approached  $22  million  in 
additional  costs  for  nonresponse  follow-up,  based  on  our  estimate,  and 
confused  respondents.  An  additional  challenge  was  that  some  of  the  maps 
enumerators  used  to  help  them  find  addresses  during  nonresponse  follow¬ 
up  contained  inaccuracies. 


*  The  initial  mail  response  rate  is  calculated  as  a  percentage  of  all  forms  in  the  mail-back 
universe  from  which  the  bureau  received  a  questionnaire.  It  factors  in  housing  units  that  are 
discovered  to  be  nonexistent  or  unoccupied  during  nonresponse  follow-up.  The  bureau 
uses  this  percentage  as  an  indicator  of  its  nonresponse  follow-up  workload.  This  differs 
from  the  mail  return  rate  which  the  bureau  uses  as  a  measure  of  public  cooperation.  It  is  the 
percentage  of  forms  the  bureau  receives  from  occupied  housing  units  in  the  mail-back 
universe  and  is  calculated  after  the  bureau  completes  the  enumeration  process. 
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Fourth,  the  bureau’s  stretch  goals  appeared  to  produce  mixed  results.  On 
the  one  hand,  on  the  basis  of  our  survey  of  local  census  office  managers, 
we  estimate  that  about  41  percent  of  managers  believed  scheduling 
pressures  had  little  or  no  impact  on  the  quality  of  the  nonresponse  follow¬ 
up  operation.  Another  17  percent  of  managers  believed  that  such  pressure 
had  a  positive  or  significantly  positive  impact.  On  the  other  hand,  about  40 
percent  of  the  local  census  office  managers  believed  that  scheduling 
pressures  during  nonresponse  follow-up  had  a  negative  or  significantly 
negative  impact  on  the  quality  of  the  operation.  A  common  concern 
appeared  to  be  that  scheduling  pressures  created  a  culture  that  emphasized 
quantity  over  quality. 

One  indicator  of  the  quality  of  nonresponse  follow-up  is  the  completeness 
of  the  data  collected  by  enumerators.  During  nonresponse  follow-up,  a 
small  number  of  local  census  offices — in  some  highly  publicized 
incidents — improperly  collected  less  complete  data  and  took  other  short¬ 
cuts  (which  the  bureau  took  steps  to  rectify).  Nationally,  however,  our 
analysis  of  bureau  data  found  that  those  offices  that  completed  their 
follow-up  workloads  faster  than  the  others  did  not  collect  larger  quantities 
of  less  complete  data,  such  as  partial  interviews. 

Finally,  questions  surround  the  extent  to  which  certain  reinterview 
procedures  aimed  at  detecting  enumerator  fraud  and  other  quality 
problems  were  implemented  throughout  the  entire  nonresponse  follow-up 
operation  as  intended.  The  decision  to  subject  enumerators’  work  to  these 
procedures  was  at  the  discretion  of  local  census  personnel.  Fifty-two  local 
census  offices  (about  10  percent  of  all  local  offices)  did  not  conduct  any 
reinterviews  after  a  random  check  of  enumerators’  initial  work.  A  senior 
bureau  quality  assurance  official  expressed  concerns  about  the  adequacy 
of  quality  assurance  coverage  toward  the  end  of  nonresponse  follow-up  at 
these  offices  because  once  random  reinterviews  were  completed  at  those 
offices,  there  were  no  additional  checks  specifically  designed  to  detect 
fabricated  data. 

In  light  of  these  challenges,  as  the  bureau  plans  for  the  next  national  head 
count  in  2010,  we  recommend  that  the  Secretary  of  Commerce  ensure  that 
the  bureau 

•  develop  and  refine  the  lessons  learned  from  the  nonresponse  follow-up 
effort  and  apply  them  to  the  planning  efforts  for  2010; 
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•  assess,  to  the  extent  practicable,  why  people  who  were  aware  of  the 
census  did  not  participate,  and  develop  appropriate  marketing 
strategies; 

•  develop  and  test  options  that  could  generate  more  current  nonresponse 
follow-up  address  lists  and  maps; 

•  ensure  that  the  bureau’s  procedures  and  incentives  for  the  timely 
completion  of  nonresponse  follow-up  emphasize  the  collection  of 
quality  data  and  proper  enumeration  techniques  as  much  as  speed;  and 

•  ensure  that  the  bureau’s  reinterview  procedures,  as  implemented,  are 
sufficient  for  consistently  and  reliably  detecting  potential  quality 
problems  throughout  the  full  duration  of  enumerators’  employment  on 
nonresponse  follow-up. 

The  Secretary  of  Commerce  forwarded  written  comments  from  the  Bureau 
of  the  Census  on  a  draft  of  this  report.  The  bureau  concurred  with  all  five 
of  our  recommendations.  The  bureau  also  clarified  several  key  points  and 
provided  additional  information  and  perspective,  which  we  incorporated  in 
our  report  as  appropriate. 


Background 


In  conducting  nonresponse  follow-up,  the  bureau  has  historically  faced  the 
twin  challenge  of  (1)  collecting  quality  data  (by  obtaining  complete  and 
accurate  information  directly  from  household  members)  while  (2)  finishing 
the  operation  on  schedule,  before  error  rates  can  increase  as  people  move 
or  have  trouble  recalling  who  was  living  at  their  homes  on  Census  Day 
(April  1),  as  well  as  keeping  subsequent  operations  on-track.  Nonresponse 
follow-up  was  scheduled  to  begin  on  April  27,  2000,  and  end  10  weeks  later, 
on  July  7,  2000. 

Local  census  offices  generally  finished  their  nonresponse  follow-up 
workloads  ahead  of  the  bureau’s  10-week  schedule.^  As  shown  in  figure  1, 
of  the  bureau’s  511  local  offices  in  the  50  states,  463  (91  percent)  finished 
nonresponse  follow-up  by  the  end  of  the  eighth  week  of  the  operation, 
consistent  with  the  bureau’s  internal  stretch  goals.  Moreover,  nine  local 
offices  completed  their  workloads  in  as  little  as  5  weeks  or  less. 


^  The  completion  time  excludes  certain  follow-up  activities  conducted  after  the  bureau 
finished  its  initial  workload. 
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Figure  1:  Local  Census  Offices  Generally  Completed  Nonresponse  Follow-up  Ahead 
of  Schedule 


Source:  GAO  analysis  of  Census  Bureau  data. 

The  timely  completion  of  nonresponse  follow-up  in  2000  stands  in  sharp 
contrast  to  the  bureau’s  experience  during  the  1990  Census.  As  shown  in 
figure  2,  at  the  end  of  the  6-week  scheduled  time  frame  for  nonresponse 
follow-up  during  the  1990  Census,  the  bureau  had  not  completed  the 
operation.  In  fact,  as  of  two  days  prior  to  the  scheduled  end  date,  just  two 
local  census  offices  had  completed  the  operation  and  the  bureau  had  only 
completed  about  72  percent  of  its  34  million  household  follow-up 
workload.  It  took  the  bureau  a  total  of  14  weeks  to  complete  the  entire 
operation.  By  comparison,  as  noted  above,  the  bureau  completed 
nonresponse  follow-up  in  less  than  10  weeks  during  the  2000  Census. 

Figure  2  also  highhghts  the  drop-off  in  production  that  occurs  during  the 
later  weeks  of  nonresponse  follow-up.  According  to  the  bureau,  the 
decline  occurs  because  unresolved  cases  at  the  end  of  nonresponse  follow¬ 
up  are  typically  the  most  difficult  to  reach,  either  because  they  are 
uncooperative  or  are  rarely  at  home  and  are  imknown  to  neighbors. 
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Figure  2:  Nonresponse  Follow-up  Workload  Completion  Rates  for  the  1990  and  2000 
Censuses 

Percentage  of  nonresponse  follow-up  workload  completed 


Scheduled  end  Scheduled  end 

of  nonreponse  of  nonreponse 

follow-up  1990  follow-up  2000 

Week  of  nonresponse  follow-up 

-  2000  completion  rate 

.  1 990  completion  rate 


Source:  GAO  analysis  of  Census  Bureau  data. 


Scope  and 
Methodology 


To  meet  our  objectives,  we  used  a  combination  of  approaches  and  methods 
to  examine  the  conduct  of  nonresponse  follow-up.  These  included 
statistical  analyses;  interviews  with  key  bureau  headquarters  officials, 
regional  census  center  officials,  and  local  census  office  managers  and  staff; 
observations  of  local  census  offices’  nonresponse  follow-up  operations; 
and  reviews  of  relevant  documentation. 

To  examine  the  factors  that  contributed  to  the  timely  completion  of 
nonresponse  follow-up,  we  interviewed  local  census  office  managers  and 
other  supervisory  staff  at  60  local  census  offices  we  visited  across  the 
country.  These  offices  generally  faced  specific  enumeration  challenges 
when  nonresponse  follow-up  began  in  late  April,  and  were  thus  prone  to 
operational  problems  that  could  affect  data  quality  (see  app.  I  for  a 
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complete  list  of  the  offices  we  visited).  Specifically,  these  offices  had  (1)  a 
larger  nonresponse  follow-up  workload  than  initially  planned;  (2)  multiple 
areas  that  were  relatively  hard-to-enumerate,  such  as  non-English-speaking 
groups;  and  (3)  difficulties  meeting  their  enumerator  recruiting  goals. 
During  these  visits,  which  took  place  in  June  and  July  2000,  we  also 
observed  office  operations  to  see  how  office  staff  were  processing 
questionnaires;  at  12  of  these  offices  we  attended  enumerator  training;  and 
at  31  offices  we  reviewed  key  reinterview  documents  in  a  given  week 
during  nonresponse  follow-up.  The  local  census  offices  we  visited 
represent  a  mix  of  urban,  suburban,  and  rural  locations.  However,  because 
they  were  judgmentally  selected,  our  findings  from  these  visits  cannot  be 
projected  to  the  universe  of  local  census  offices. 

To  obtain  a  broader  perspective  of  the  conduct  of  nonresponse  follow-up, 
we  used  the  results  of  our  survey  of  a  stratified  random  sample  of 
managers  at  250  local  census  offices.  The  survey — ^which  asked  these 
managers  about  the  implementation  of  a  number  of  key  field  operations — 
is  generalizable  to  the  511  local  census  offices  located  in  the  50  states.  ^  We 
obtained  responses  from  managers  at  236  local  census  offices  (about  a  94 
percent  overall  response  rate).  All  reported  percentages  are  estimates 
based  on  the  sample  and  are  subject  to  some  sampling  error  as  well  as 
nonsampling  error.  In  general,  percentage  estimates  in  this  report  for  the 
entire  sample  have  confidence  intervals  ranging  from  about  ±  4  to  ±  5 
percentage  points  at  the  95  percent  confidence  interval.  In  other  words,  if 
all  managers  in  our  local  census  office  population  had  been  surveyed,  the 
chances  are  95  out  of  100  that  the  result  obtained  would  not  differ  from  our 
sample  estimate  in  the  more  extreme  cases  by  more  than  ±  5  percent. 


^  Our  analysis  did  not  include  nine  local  census  offices  located  in  Puerto  Rico. 


Page  8 


GAO-02-196  2000  Census  Best  Practices  and  Lessons  Learned 


To  examine  whether  the  pace  of  nonresponse  follow-up  was  associated 
with  the  collection  of  less  complete  data,  in  addition  to  the  efforts 
described  above,  we  analyzed  bureau  data  on  the  weekly  progress  of 
nonresponse  follow-up.  Specific  measures  we  analyzed  included  the  time 
it  took  local  census  offices  to  finish  nonresponse  follow-up  and  the 
proportion  of  their  cases  completed  by  (1)  “close-out”  interviews,  where 
questionnaires  only  contain  basic  information  on  the  status  of  the  housing 
unit  (e.g.,  whether  it  was  occupied),  or  (2)  “partial”  interviews,  which 
contain  more  information  than  a  close-out  interview  but  are  still  less  than 
complete.  The  completeness  of  the  data  collected  by  enumerators  is  one 
measure  of  the  quality  of  nonresponse  follow-up,  and  these  two  measures 
were  the  best  indicators  of  completeness  available  from  the  database.  We 
included  data  from  the  511  offices  located  in  the  50  states  and  controlled 
for  enumeration  difficulty  using  an  index  measure  developed  by  the 
bureau.^  We  did  not  include  any  outliers  that  the  bureau  identified  as 
erroneous  (for  example,  outliers  resulting  from  coding  errors).® 

We  did  our  audit  work  at  the  local  census  offices  identified  in  appendix  I 
and  their  respective  regional  census  centers;  bureau  headquarters  in 
Suitland,  Maryland;  and  Washington,  DC,  from  March  2000  through 
September  2001.  Our  work  was  done  in  accordance  with  generally 
accepted  government  auditing  standards. 

We  requested  comments  on  a  draft  of  this  report  from  the  Secretary  of 
Commerce.  On  January  10,  2002,  the  Secretary  forwarded  the  bureau’s 
written  comments  on  the  draft  (see  app.  II)  which  we  address  at  the  end  of 
this  report. 


^  The  index  measure,  or  “hard-to-count  score,”  was  based  on  variables  contained  in  the  1990 
Data  for  Census  2000  Planning  Database,  such  as  the  percent  of  households  with  no  adult 
who  speaks  Enghsh  well. 

®  Of  the  511  local  offices,  3  were  not  included  in  the  analysis  of  partial  interviews  and  12 
were  not  included  in  the  analysis  of  closeout  interviews  because  the  bureau  identified  their 
values  for  these  variables  as  erroneous  due  to  coding  errors. 
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The  Bureau  Used  an 
Aggressive  Outreach 
and  Promotion 
Campaign  and  Other 
Strategies  to  Boost  the 
Mail  Response  Rate  but 
Public  Cooperation 
Remains  Problematic 


Key  to  the  bureau’s  timely  completion  of  nonresponse  follow-up  in  2000 
was  a  higher  than  expected  initial  mail  response  rate  that  decreased  the 
bureau’s  follow-up  workload.  In  addition  to  reducing  the  staff,  time,  and 
money  required  to  complete  the  census  count,  the  bureau’s  past  experience 
and  evaluations  suggest  that  the  quality  of  data  obtained  from 
questionnaires  returned  by  mail  is  better  than  the  data  collected  by 
enumerators. 

To  help  raise  the  mail  response  rate,  the  bureau  (1)  hired  a  consortium  of 
private-sector  advertising  agencies,  led  by  Young  &  Rubicam,  to  develop  a 
national,  multimedia  paid  advertising  program,  and  (2)  partnered  with  local 
governments,  community  groups,  businesses,  nongovernmental 
organizations,  and  other  entities  to  promote  the  census  on  a  grassroots 
basis  (we  discuss  the  bureau’s  partnership  program  in  more  detail  in  our 
August  2001  report).®  The  outreach  and  promotion  campaign  encouraged 
people  to  complete  their  census  questiormaires  by  conveying  the  message 
that  census  participation  helped  their  communities.  The  bureau  also  helped 
boost  the  mail  response  rate  by  using  simplified  questionnaires,  which  was 
consistent  with  our  past  suggestions,^  and  by  developing  more  ways  to 
respond  to  the  census,  such  as  using  the  Internet. 


The  bureau  achieved  an  initial  mail  response  rate  of  about  64  percent, 
which  was  about  3  percentage  points  higher  than  the  61  percent  response 
rate  the  bureau  expected  when  plarming  for  nonresponse  follow-up.®  This, 
in  turn,  resulted  in  a  nonresponse  follow-up  workload  of  about  42  million 
housing  units,  which  was  about  4  million  fewer  housing  units  than  the 
bureau  would  have  faced  under  its  planning  assumption  of  a  61  percent 
mail  response  rate. 


®  2000  Census:  Review  of  Partnership  Program  Highlights  Best  Practices  for  Future 
Operations  (GAO-01-579,  Aug.  20,  2001). 

^  See  for  example,  Decennial  Census:  1990  Results  Show  Need  for  Fundamental  Reform 
(GAO/GGD-92-94,  June  9,  1992). 

*  For  the  2000  Census,  the  bureau  used  what  it  refers  to  as  an  “initial  response  rate”  to 
provide  a  measure  of  the  scope  of  the  nonresponse  follow-up  operation.  This  initial 
response  rate  is  defined  as  the  percentage  of  aU  questionnaires  that  are  completed  and 
returned  by  April  18,  2000.  The  rate  includes  the  number  of  questionnaires  that  are  mailed 
back,  transmitted  via  the  Internet,  or  completed  over  the  telephone  through  the  bureau’s 
Telephone  Questionnaire  Assistance  program.  It  also  includes  Be  Counted  Forms  that  have 
census  identification  numbers.  On  September  19,  2000,  the  bureau  announced  that  it  had 
achieved  a  final  mail-back  response  rate  of  67  percent. 
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In  addition  to  surpassing  its  national  response  rate  goals,  the  bureau 
exceeded  its  own  expectations  at  the  local  level.  Of  the  511  local  census 
offices,  378  (74  percent)  met  or  exceeded  the  bureau's  expected  response 
rate.  In  so  doing,  these  offices  reduced  their  nonresponse  follow-up 
workloads  from  the  expected  levels  by  between  54  and  58,329  housing 
units.  The  remaining  133  offices  (26  percent)  did  not  meet  their  expected 
response  rate  and  the  workload  at  these  offices  increased  from  their 
expected  levels  by  between  279  and  33,402  housing  units. 


Securing  Public 
Participation  While 
Controlling  Costs  Remains  a 
Considerable  Challenge  for 
the  2010  Census 


The  bureau’s  success  in  surpassing  its  response  rate  goals  was  noteworthy 
given  the  formidable  societal  challenges  it  faced.  These  challenges 
included  attitudinal  factors  such  as  concerns  over  privacy,  and 
demographic  trends  such  as  more  complex  living  arrangements.  However, 
as  the  bureau  plans  for  the  next  census  in  2010,  it  faces  the  difficulty  of 
boosting  public  participation  while  keeping  costs  manageable. 


As  we  noted  in  our  December  2001  report,  although  the  bureau  achieved 
similar  response  rates  in  1990  and  2000  (65  percent  in  1990  and  64  percent 
in  2000),  the  bureau  spent  far  more  money  on  outreach  and  promotion  in 
2000:  about  $3.19  per  household  in  2000  compared  to  $0.88  in  1990  (in 
constant  fiscal  year  2000  dollars),  an  increase  of  260  percent.®  Moreover, 
the  societal  challenges  the  bureau  encountered  in  1990  and  2000  will 
probably  be  more  complex  in  2010,  and  simply  staying  on  par  with  the  2000 
response  rate  will  likely  require  an  even  greater  investment  of  bureau 
resources. 


Further,  while  the  mail  response  rate  provides  a  direct  indication  of  the 
nonresponse  workload,  it  is  an  imperfect  measure  of  public  cooperation 
with  the  census  as  it  is  calculated  as  a  percentage  of  all  forms  in  the  mail- 
back  universe  from  which  the  bureau  received  a  questionnaire.  Because 
the  mail-back  universe  includes  housing  units  that  the  bureau  determines 
are  nonexistent  or  vacant  during  nonresponse  follow-up,  a  more  precise 
measure  of  public  cooperation  is  the  mail  return  rate,  which  excludes 
vacant  and  nonexistent  housing  units.  According  to  preliminary  bureau 
data,  the  mail  return  rate  for  the  2000  Census  was  72  percent,  a  decline  of  2 
percentage  points  from  the  74  percent  mail  return  rate  the  bureau  achieved 
in  1990.  As  shown  in  figure  3,  in  2000,  the  bureau  reduced,  but  did  not 


^  2000  Census:  Significant  Increase  in  Cost  Per  Housing  Unit  Compared  to  1990  Census 
(GAO-02-31,  Dec.  11,  2001). 


Page  11 


GAO-02-196  2000  Census  Best  Practices  and  Lessons  Learned 


reverse,  the  steady  decline  in  public  cooperation  that  has  occurred  with 
each  decennial  census  since  the  bureau  first  initiated  a  national  mail- 
out/mail-back  approach  in  1970.  Bureau  officials  said  they  would  further 
examine  the  reasons  for  the  decline  in  the  return  rate  as  part  of  its  Census 
2000  evaluations. 


Figure  3:  Public  Cooperation  with  the  Census  Has  Steadily  Declined 
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Source:  GAO  analysis  of  Census  Bureau  data. 

In  addition,  as  shown  in  figure  4,  the  results  to  date  show  that  just  three 
states  increased  their  mail  return  rates  compared  to  the  1990  Census. 
Overall,  preliminary  bureau  data  shows  the  change  in  mail  return  rates 
from  1990  through  2000  ranged  from  an  increase  of  about  1  percentage 
point  in  Massachusetts  and  California  to  a  decline  of  about  9  percentage 
points  in  Kentucky. 
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Figure  4:  2000  Census  Return  Rates  Declined  in  Most  States  Compared  to  1990 
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Source:  GAO  analysis  of  preliminary  Census  Bureau  data. 
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The  bureau’s  outreach  and  promotion  efforts  will  also  face  the  historical 
hurdle  of  bridging  the  gap  that  exists  between  the  public’s  awareness  of  the 
census  on  the  one  hand,  and  its  motivation  to  respond  on  the  other. 
Various  polls  conducted  for  the  2000  Census  suggested  that  the  public’s 
awareness  of  the  census  was  over  90  percent;  and  yet,  as  noted  earlier,  the 
actual  return  rate  was  much  lower — 72  percent  of  the  nation’s  households. 
The  bureau  faced  a  similar  issue  in  1990  when  93  percent  of  the  public 
reported  being  aware  of  the  census,  but  the  return  rate  was  74  percent.  In 
our  previous  work,  we  noted  that  closing  this  gap  would  be  a  significant 
challenge  for  the  bureau,  and  as  the  bureau  plans  for  the  2010  Census,  it 
will  be  important  for  it  to  explore  approaches  that  more  effectively  convert 
the  public’s  awareness  of  the  census  into  a  willingness  to  respond. 


Flexible  Human 
Capital  Strategies 
Helped  the  Bureau 
Meet  Its  Recruitment 
Goals 


A  second  factor  that  was  instrumental  to  the  operational  success  of 
nonresponse  follow-up  was  an  ample  and  sufficiently  skilled  enumerator 
workforce.  Based  on  anticipated  turnover  and  the  expected  workload  to 
carry  out  its  four  largest  field  data  collection  operations — of  which 
nonresponse  follow-up  was  the  largest — the  bureau  set  a  recruitment  goal 
of  2.4  million  qualified  applicants.^  In  addition  to  the  sheer  volume  of 
recruits  needed,  the  bureau's  efforts  were  complicated  by  the  fact  that  it 
was  competing  for  employees  in  a  historically  tight  national  labor  market. 
Nevertheless,  when  nonresponse  follow-up  began  on  April  27,  the  bureau 
had  recruited  over  2.5  million  qualified  applicants. 


The  bureau  surmounted  its  human  capital  challenge  with  an  aggressive 
recruitment  strategy  that  helped  make  the  bureau  a  more  attractive 
employer  to  prospective  candidates  and  ensured  a  steady  stream  of 
applicants.  Key  ingredients  of  the  bureau’s  recruitment  efforts  included  the 
following: 

1.  A  geographic  pay  scale  with  wages  set  at  65  to  75  percent  of  local 
prevailing  wages  (from  about  $8.25  to  $18.50  per  hour  for 
enumerators).  The  bureau  also  used  its  flexibility  to  raise  pay  rates  for 
those  census  offices  that  were  encountering  recruitment  difficulties. 


“  2000  Census:  Preparations  for  Dress  Rehearsal  Leave  Many  Unanswered  Questions 
(GAO/GGD-98-74,  Mar.  26,  1998). 

“  The  bureau  later  adjusted  its  qualified  applicant  goal  to  2.1  million  based  on  the  actual 
nonresponse  follow-up  workload. 
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For  example,  a  manager  at  one  of  the  Charlotte  region’s  local  census 
offices  told  us  that  the  office  was  having  difficulty  obtaining  needed 
staff  in  part  because  census  wages  were  uncompetitive.  According  to 
this  manager,  the  region  approved  a  pay  increase  for  the  office’s 
enumerators  and  office  clerks,  which  helped  the  office  obtain  staff.  In 
all,  when  nonresponse  follow-up  began,  the  bureau  raised  pay  rates  for 
field  staff  at  eight  local  offices  to  address  those  offices’  recruiting 
challenges. 

2.  Partnerships  with  state,  local,  and  tribal  governments,  community 
groups,  and  other  organizations  to  help  recruit  employees  and  provide 
free  facilities  to  test  applicants.  For  example.  Clergy  United,  an 
organization  representing  churches  in  the  Detroit  metropolitan  area, 
provided  space  for  testing  census  job  applicants  in  December  1998. 

The  organization  even  conducted  pre-tests  several  days  before  each 
bureau-administered  test  so  those  applicants  could  familiarize 
themselves  with  the  testing  format. 

3.  A  recruitment  advertising  campaign,  which  totaled  over  $2.3  million, 
that  variously  emphasized  the  ability  to  earn  good  pay,  work  flexible 
hours,  learn  new  skills,  and  do  something  important  for  one’s 
community.  Moreover,  the  advertisements  were  in  a  variety  of 
languages  to  attract  different  ethnic  groups,  and  were  also  targeted  to 
different  races,  senior  citizens,  retirees,  and  people  seeking  part-time 
employment.  The  bureau  advertised  using  traditional  outlets  such  as 
newspaper  classified  sections,  as  well  as  more  novel  media  including 
Internet  banners  and  messages  on  utility  and  credit  card  bills. 

4.  Obtaining  exemptions  from  the  majority  of  state  governments  so  that 
individuals  receiving  Temporary  Assistance  for  Needy  Families, 
Medicaid,  and  selected  other  types  of  public  assistance  would  not  have 
their  benefits  reduced  when  earning  census  income,  thus  making 
census  jobs  more  attractive.  At  the  start  of  nonresponse  follow-up,  44 
states  and  the  Virgin  Islands  had  granted  an  exemption  for  one  or  more 
of  these  programs. 

5.  Encouraging  local  offices  to  continue  their  recruiting  efforts 
throughout  nonresponse  follow-up,  regardless  of  whether  offices  had 
met  their  recruiting  goals,  to  ensure  a  steady  stream  of  available 
applicants. 
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The  bureau  matched  these  initiatives  with  an  ongoing  monitoring  effort 
that  enabled  bureau  officials  to  rapidly  respond  to  recruiting  difficulties. 
For  example,  during  the  last  2  weeks  of  April,  the  bureau  mailed  over  5 
million  recruiting  postcards  to  Boston,  Charlotte,  and  other  locations 
where  it  found  recruitment  efforts  were  lagging. 

Based  on  the  results  of  our  local  census  office  visits,  it  is  clear  that  the 
bureau’s  human  capital  strategy  had  positive  outcomes.  Of  the  60  local 
census  offices  we  visited,  officials  at  59  offices  provided  useable  responses 
to  our  question  about  whether  their  offices  had  the  type  of  staff  they 
needed  to  conduct  nonresponse  follow-up,  including  staff  with  particular 
language  skills  to  enumerate  in  targeted  areas.  Officials  at  54  of  the  59 
offices  said  they  had  the  type  of  staff  they  needed  to  conduct  nonresponse 
follow-up.  For  example,  officials  in  the  Boston  North  office  said  they  hired 
enumerators  who  spoke  Japanese,  Vietnamese,  Portuguese,  Spanish, 
French,  Russian,  and  Chinese,  while  Pittsburgh  office  officials  said  they 
had  enumerators  that  knew  sign  language  to  communicate  with  deaf 
residents. 

Managers  at  local  census  offices  we  surveyed  provided  additional 
perspective  on  recruiting  needed  field  staff.  As  shown  in  figure  5,  30 
percent  of  the  respondents  believed  that  the  bureau’s  ability  to  recruit  and 
hire  high-quality  field  staff  needed  no  improvements.  While  managers  at  52 
percent  of  the  local  offices  commented  that  some  improvement  to  the 
recruiting  and  hiring  process  was  needed  and  another  17  percent 
commented  that  a  significant  amount  of  improvement  was  needed,  their 
suggestions  varied.  Managers’  suggestions  generally  related  to  various 
hiring  practices,  such  as  a  greater  use  of  face-to-face  interviews  to  select 
managers  at  local  census  offices  and  earlier  recruitment  advertising. 


At  one  of  the  local  census  offices  we  visited,  we  were  unable  to  obtain  a  useable  response 
to  this  question  generally  because  the  local  census  office’s  managers  were  unavailable 
during  the  time  of  our  review. 
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Figure  5:  Local  Managers’  Perceptions  of  Recruiting  and  Hiring 
100  Percentage  of  local  census  offices 
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Source:  GAO  survey  of  local  census  office  managers. 

Once  nonresponse  follow-up  began,  bureau  officials  tracked  production 
rates  as  the  primary  measure  of  whether  local  offices  had  met  their  staffing 
goals.  For  example,  bureau  officials  said  that  both  bureau  headquarters 
and  regional  census  center  staff  monitored  local  census  offices’  production 
daily.  If  an  office  was  not  meeting  its  production  goals,  bureau 
headquarters  officials  said  they  worked  with  regional  census  personnel, 
who  in  turn  worked  with  the  local  census  office  manager,  to  determine  the 
reasons  for  the  shortfall  and  the  actions  necessary  to  increase  production. 
Possible  actions  included  bringing  in  enumerators  from  neighboring  local 
census  offices. 

Overall,  preliminary  bureau  data  shows  that  about  500,000  enumerators 
worked  on  nonresponse  follow-up.  Nationally,  the  bureau  established  a 
hiring  goal  of  292,000  enumerator  positions  for  nonresponse  follow-up. 
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which  represented  two  people  working  approximately  25  hours  per  week 
for  each  position  and  assumed  100  percent  turnover,  according  to  bureau 
officials.  The  bureau  has  not  yet  analyzed  how  many  enumerators  charged 
at  least  25  hours  per  week  during  nonresponse  follow-up.  Moreover, 
according  to  a  senior  bureau  official,  the  bureau  has  not  decided  whether  it 
will  do  such  an  analysis  for  2010  planning  purposes.  According  to  this 
official,  because  the  bureau  hired  about  500,000  enumerators  and 
completed  the  operation  a  week  ahead  of  schedule,  they  believe  the  bureau 
generally  met  its  hiring  goal. 


Local  Census  Offices 
Planned  in  Advance  for 
Specific  Enumeration 
Challenges 


A  third  factor  that  contributed  to  the  timely  completion  of  nonresponse 
follow-up  was  preparing  in  advance  for  probable  enumeration  challenges. 
To  do  this,  the  bureau  called  on  local  census  offices  and  their  respective 
regional  census  centers  to  develop  action  plans  that,  among  other  things, 
identified  hard-to-enumerate  areas  within  their  jurisdictions,  such  as 
immigrant  neighborhoods,  and  propose  strategies  for  dealing  with  those 
challenges.  These  strategies  included  such  methods  as  paired/team 
enumeration  for  high-crime  areas,  and  hiring  bilingual  enumerators.  While 
this  early  planning  effort  helped  local  census  offices  react  to  a  variety  of 
enumeration  challenges,  the  currency  and  accuracy  of  the  nonresponse 
follow-up  address  lists  and  maps  remained  problematic  for  a  number  of 
local  census  offices. 


Most  Local  Offices  Used 
Action  Plans  to  Address 
Enumeration  Challenges 


Of  the  60  local  census  offices  we  visited,  officials  at  55  offices  provided 
useable  responses  to  our  question  about  how,  if  at  all,  their  offices  used 
their  action  plan  for  hard-to-enumerate  areas  during  nonresponse  follow¬ 
up.  Officials  at  51  of  55  offices  said  their  offices  used  the  strategies  in 
their  action  plan  to  address  the  enumeration  challenges  they  faced. 

At  the  offices  we  visited,  a  frequently  cited  enumeration  challenge  was 
gaining  access  to  gated  communities  or  secure  apartment  buildings. 
Officials  at  42  of  the  60  offices  we  visited  identified  this  as  a  problem.  To 
address  it,  officials  said  they  developed  partnerships  with  building 
management  and  community  leaders,  among  other  strategies.  In  an  Atlanta 


“  At  five  of  the  local  census  offices  we  visited,  we  were  unable  to  obtain  a  useable  response 
to  this  question  generally  because  local  census  office  managers  were  either  unavailable  or 
did  not  know. 
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office,  for  example,  local  officials  said  they  sent  letters  to  managers  of 
gated  communities  that  stressed  the  importance  of  the  census.  Similarly, 
officials  in  a  Chicago  office  said  they  personally  phoned  managers  of 
secure  apartment  buildings.  When  enumerators  from  a  Milwaukee  local 
census  office  encountered  problems  accessing  locked  apartment  buildings, 
local  census  officials  told  us  that  the  City  of  Milwaukee  sent  aldermen  to 
visit  the  building  managers  and  encourage  them  to  participate  in  the 
census. 

Another  common  enumeration  challenge  appeared  to  be  obtaining 
cooperation  from  residents — cited  as  a  difficulty  by  officials  at  34  of  the  60 
offices  we  visited.  One  problem  they  noted  was  obtaining  responses  to  the 
long-form  questionnaire — either  in  its  entirety  or  to  specific  items,  such  as 
income-related  questions-which,  according  to  local  census  officials,  some 
residents  found  to  be  intrusive. 

Enumerators  also  encountered  residents  who  were  unwilling  to  participate 
in  the  census  because  of  language  and  cultural  differences,  or  their  fears  of 
government.  The  bureau’s  standardized  training  for  enumerators  included 
procedures  for  handling  refusals.  Local  census  officials  encouraged  public 
participation  with  a  variety  of  approaches  as  well.  For  example,  census 
officials  in  Cleveland  and  Cincinnati  said  they  provided  additional  training 
for  enumerators  on  how  to  handle  refusals  and  practiced  what  was  taught 
in  mock  interviews.  Officials  in  other  census  offices  said  they  partnered 
with  local  community  leaders  who  subsequently  helped  reach  out  to  hard- 
to-enumerate  groups,  hired  people  who  were  bilingual  or  otherwise  trusted 
and  known  by  residents,  and  held  media  campaigns.  Overall,  according  to 
bureau  data,  close  to  470,000  households  of  the  approximately  42  million 
making  up  the  nonresponse  follow-up  workload  (about  1  percent),  refused 
to  participate  in  the  census. 


The  Accuracy  and  Currency 
of  Nonresponse  Follow-up 
Address  Lists  and  Maps 
Appeared  to  Be  Problematic 


Of  the  60  local  census  offices  we  visited,  officials  at  52  offices  provided 
useable  responses  to  our  question  about  whether  their  offices’ 
nonresponse  follow-up  address  list  reflected  the  most  accurate  and  current 
information.^^  Officials  at  21  of  the  52  offices  said  that  their  lists  generally 
were  not  accurate  and  current.  Nationwide,  as  shown  in  figure  6,  based  on 


“  At  eight  local  census  offices  we  visited,  we  were  unable  to  obtain  a  useable  response  to 
this  question  generally  because  local  census  office  managers  were  either  unavailable  or  did 
not  know. 
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our  survey  of  local  census  office  managers,  we  estimate  that  managers  at 
approximately  50  percent  of  local  census  offices  believed  that  some 
improvement  was  needed  in  the  accuracy  of  address  lists  for  nonresponse 
follow-up.  We  estimated  that  managers  at  about  22  percent  of  local  census 
offices  believed  that  a  significant  amount  of  improvement  was  needed. 


Figure  6:  Local  Managers’  Perceptions  of  the  Accuracy  of  Nonresponse  Follow-up 
Address  Lists 
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Source:  GAO  survey  of  local  census  office  managers. 

Among  the  more  frequent  problems  managers  cited  were  duplicate 
addresses  and  changes  not  being  made  from  prior  operations.  For 
example,  at  a  local  census  office  in  the  Seattle  region,  managers  said  that 
some  addresses  were  residences  or  businesses  that  had  been  gone  for  10-15 
years  and  should  have  been  deleted  in  previous  census  operations  but  were 
not. 
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Local  census  officials  we  visited  cited  problems  with  the  accuracy  of  the 
census  maps  as  well.  Of  the  60  local  census  offices  we  visited,  officials  at  58 
offices  provided  useable  responses  to  our  question  about  whether  the  most 
accurate  and  current  information  was  reflected  on  the  nonresponse  follow¬ 
up  maps.  Officials  at  about  a  third  of  local  census  offices — 21  of  58 
offices — said  the  nonresponse  follow-up  maps  did  not  reflect  the  most 
accurate  and  current  information. 

Further,  as  shown  in  figure  7,  based  on  our  survey  of  local  census  office 
managers,  at  about  41  percent  of  the  offices,  managers  believed  that  some 
improvement  was  needed  in  maps  for  nonresponse  follow-up.  At  about  23 
percent  of  the  offices,  managers  believed  that  a  significant  amount  of 
improvement  was  needed  in  these  maps. 


Figure  7:  Local  Managers’  Perceptions  of  the  Accuracy  of  Maps 
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Source:  GAO  survey  of  local  census  office  managers. 


“  At  two  of  the  local  census  offices  we  visited,  we  were  unable  to  obtain  a  useable  response 
to  this  question  generally  because  local  census  office  managers  were  either  unavailable  or 
did  not  know. 
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Managers  who  commented  that  improvements  were  needed  to  the 
nonresponse  follow-up  maps  said  the  maps  were  difficult  to  use,  not 
updated  from  prior  operations,  and  contained  errors.  For  example,  an 
official  at  a  local  census  office  in  the  Atlanta  region  said  that  some  roads  on 
the  map  did  not  exist  or  were  not  oriented  correctly  on  the  census  maps. 
To  address  this  difficulty,  local  office  staff  purchased  commercial  maps  or 
used  the  Internet  to  help  them  locate  some  housing  units. 

The  bureau  developed  its  master  address  list  and  maps  using  a  series  of 
operations  that  made  incremental  updates  designed  to  continuously 
improve  the  completeness  and  accuracy  of  the  master  address  file  and 
maps.  A  number  of  these  updates  occurred  during  nonresponse  follow-up 
when  enumerators  encountered,  for  example,  nonexistent  or  duplicate 
housing  units,  or  units  that  needed  to  be  added  to  the  address  list.  As  a 
result,  the  bureau  was  expecting  some  discrepancies  between  the 
nonresponse  follow-up  address  list  and  what  enumerators  found  in  the 
field  when  they  went  door-to-door,  which  could  account  for  some  of  the 
local  census  officials’  perceptions. 

Another  factor  that  affected  the  currency  of  the  nonresponse  follow-up 
address  list  was  the  cut-off  date  for  mail-back  responses.  The  bureau  set 
April  11,  2000,  as  the  deadline  for  mail-back  responses  for  purposes  of 
generating  the  address  list  for  nonresponse  follow-up.  In  a  subsequent  late 
mail  return  operation,  the  bureau  updated  its  field  follow-up  workload  by 
removing  those  households  for  which  questionnaires  were  received  from 
April  11  through  April  18.  However,  according  to  bureau  officials,  the 
bureau  continued  to  receive  questionnaires,  in  part  because  of  an 
unexpected  boost  from  its  outreach  and  promotion  campaign.  For 
example,  by  April  30 — less  than  2  weeks  after  the  bureau  removed  the  late 
mail  returns  that  it  had  checked-in  as  of  April  18-the  bureau  received 
773,784  additional  questionnaires.  Bureau  headquarters  officials  told  us  it 
was  infeasible  to  remove  the  late  returns  from  the  nonresponse  follow-up 
address  lists  and  thus,  enumerators  needed  to  visit  these  households. 

The  cost  of  these  visits  approached  $22  million,  based  on  our  earlier 
estimate  that  a  1-percentage  point  increase  in  workload  could  add  at  least 
$34  million  in  direct  salary,  benefits,  and  travel  costs  to  the  price  tag  of 
nonresponse  follow-up.^®  In  addition,  the  bureau’s  data  processing  centers 
then  had  to  reconcile  the  duplicate  questionnaires.  According  to  officials 


“  GAO/GGD-OO-6,  December  14,  1999. 
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at  some  local  offices  we  visited,  the  visits  to  households  that  had  already 
responded  confused  residents  who  questioned  why  enumerators  came  to 
collect  information  from  them  after  they  had  mailed  back  their  census 
forms. 


The  Bureau’s  Stretch 
Goals  to  Complete 
Nonresponse  Follow¬ 
up  May  Have  Produced 
Mixed  Results 


To  help  ensure  that  local  census  offices  completed  nonresponse  follow-up 
on  schedule,  the  bureau  developed  ambitious  interim  stretch  goals.  These 
goals  called  on  local  census  offices  to  finish  80  percent  of  their 
nonresponse  follow-up  workload  within  the  first  4  weeks  of  the  operation 
and  be  completely  finished  by  the  end  of  the  eighth  week.  Under  the 
bureau’s  master  schedule,  local  census  offices  had  10  weeks  to  complete 
the  operation. 


Local  Census  Office 
Managers  Cited  Both 
Positive  and  Negative 
Effects  of  the  Nonresponse 
Follow-up  Schedule  on  the 
Quality  of  the  Operation 


Our  survey  of  local  census  office  managers  asked  what  impact,  if  any, 
scheduling  pressures  to  complete  nonresponse  follow-up  had  on  the  quality 
of  the  operation.  On  the  one  hand,  as  shown  in  figure  8,  about  41  percent  of 
the  local  census  office  managers  believed  that  scheduling  pressures  had 
little  or  no  impact  on  the  quality  of  the  operation,  while  about  17  percent 
believed  that  such  pressure  had  a  positive  or  significantly  positive  impact. 
At  a  local  census  office  in  the  New  York  region,  for  example,  the  local 
census  office  manager  stated  that,  "pressuring  people  a  little  gave  them  the 
motivation  to  produce.”  Managers  in  local  census  offices  located  in  the 
Dallas  region  commented  that  the  schedule  “kept  people  on  their  toes  and 
caused  them  to  put  forth  their  best  effort"  and  that  it  “had  a  positive  impact, 
particularly  on  quality.” 


On  the  other  hand,  managers  at  a  substantial  number  of  local  census 
offices  had  the  opposite  view.  As  shown  in  figure  8,  about  40  percent  of  the 
respondents  believed  that  scheduling  pressure  during  nonresponse  follow¬ 
up  had  a  negative  or  significantly  negative  impact  on  the  quality  of  the 
operation. 
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Figure  8:  Local  Managers’  Views  on  the  Impact  of  Scheduling  Pressures  on  the 
Quality  of  Nonresponse  Follow-up 
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Note:  Due  to  rounding,  percentages  do  not  equal  100  percent. 

Source:  GAO  survey  of  local  census  office  managers. 

Of  those  managers  who  believed  that  the  pressure  to  complete 
nonresponse  follow-up  adversely  affected  the  quality  of  the  operation,  a 
common  perception  appeared  to  be  that  production  was  emphasized  more 
than  accuracy  and  that  the  schedule  required  local  census  offices  to  curtail 
procedures  that  could  have  improved  data  quality.  For  example,  managers 
at  some  local  census  offices  told  us  that  the  bureau’s  regional  census 
centers  encouraged  competition  between  local  census  offices  by,  among 
other  actions,  ranking  local  census  offices  by  their  progress  and 
distributing  the  results  to  local  managers.  Managers  at  some  local  census 
offices  believed  that  such  competition  fostered  a  culture  where  quantity 
was  more  important  than  quality.  As  one  manager  told  us,  the  bureau’s 
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ambitious  nonresponse  follow-up  schedule  led  the  manager  “to  put 
enormous  pressure  on  people  in  the  field  to  complete  the  operation 
quickly,  and  this  affected  the  quality  of  data.”  However,  none  of  the 
managers  we  surveyed  cited  specific  examples  of  where  comers  were  cut 
or  quality  was  compromised. 


The  Pace  of  Nonresponse 
Follow-up  Was  Not 
Associated  with  the 
Collection  of  Less  Complete 
Data 


One  measure  of  the  quality  of  nonresponse  follow-up  is  the  completeness 
of  the  data  collected  by  enumerators.  The  bureau  went  to  great  lengths  to 
obtain  complete  data  directly  from  household  members.  Bureau 
procedures  generally  called  for  enumerators  to  make  up  to  three  personal 
visits  and  three  telephone  calls  to  each  household  on  different  days  of  the 
week  at  different  times  until  they  obtained  needed  information  on  that 
household. 


However,  in  cases  where  household  members  could  not  be  contacted  or 
refused  to  answer  all  or  part  of  the  census  questionnaire,  enumerators 
were  permitted  to  obtain  data  via  proxy  (a  neighbor,  building  manager,  or 
other  nonhousehold  member  presumed  to  know  about  its  residents),  or 
collect  less  complete  data  than  called  for  by  the  census  questioimaire. 
Such  data  include  (1)  “closeout”  interviews,  where  questionnaires  only 
contain  the  information  on  the  status  of  the  housing  unit  (e.g.,  whether  or 
not  it  was  occupied),  and  the  number  of  residents  and  (2)  “partial” 
interviews,  which  contain  more  information  than  a  closeout  interview  but 
less  than  a  completed  questioimaire. 

There  were  several  well-publicized  breakdowns  in  these  enumeration 
procedures  at  a  small  number  of  local  census  offices  that  took  short  cuts  to 
complete  their  work  (which  the  bureau  later  took  steps  to  rectify). 
Nationally,  however,  our  analysis  of  bureau  data  found  no  statistically 
significant  association  between  the  week  individual  local  census  offices 
finished  their  nonresponse  follow-up  workload  and  the  percentage  of 
partial^^  or  closeout^®  interviews  they  reported,  after  controlling  for  the 
enumeration  difficulty  level  of  each  local  office’s  area^®  (at  the  time  of  our 


Results  of  regression:  t  =  -1.65;  p  =  0.10. 

Results  of  regression:  t  =  -0.44;  p  =  0.66. 

“  We  used  an  index  measure  (hard-to-count  score)  developed  by  the  bureau. 
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review,  the  bureau  did  not  have  information  on  data  collected  via  proxy 
interviews). 

Neither  did  we  find  a  statistically  significant  relationship  between  the  week 
that  local  census  offices  finished  their  nonresponse  follow-up  workload 
and  the  amount  of  residual  workload,^®  they  had,  if  any.  The  residual 
workload  consisted  of  households  that  were  part  of  the  original  follow-up 
workload,  but  from  which  the  bureau  did  not  receive  a  questionnaire  from 
the  local  census  offices,  and  thus  had  not  been  processed  through  data 
capture.  According  to  bureau  data,  519  local  offices  had  to  conduct 
residual  nonresponse  follow-up  on  121,792  households. 

Similarly,  we  did  not  find  an  association  between  week-to-week  “spikes”  in 
local  census  offices’  production  and  the  percentage  of  either  partial  or 
closeout  interview  data  reported.  Spikes  or  surges  in  production  could 
indicate  that  local  census  offices  were  cutting  comers  to  complete  their 
workloads  by  a  specific  deadline.  Nationally,  we  found  no  relationship 
between  the  number  of  questionnaires  finished  each  week  and  either  the 
percentage  of  those  finished  that  were  closeout  interviews'^  or  partial 
interviews.^^ 

Overall,  as  shown  in  figure  9,  as  nonresponse  follow-up  progressed,  the 
proportion  of  closeout  and  partial  interview  data  collected  relative  to  the 
amount  of  questionnaires  finished  remained  relatively  constant. 


Results  of  regression:  t  =  -.04;  p  =  0.97. 
Results  of  correlation:  r  =  -.08. 

^  Results  of  correlation:  r  =  -.15. 
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Figure  9:  Collection  of  Partiai  Interview  and  Cioseout  Data  Remained  Reiatively 
Constant  Throughout  Nonresponse  Foiiow-up 
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Ending  week  that  nonresponse  follow-up  completed 

Cumulative  nonresponse  follow-up  workload  finished 
Cumulative  partial  Interview  data  collected 
Cumulative  closeout  data  collected 


Note:  There  were  no  bureau  data  available  for  weeks  1  and  10.  Comparable  data  for  1990  were  not 
available  for  comparison  to  2000  results.  Percentage  of  workload  finished  is  out  of  the  total  workload; 
percentages  of  partial  interviews  and  closeouts  are  out  of  the  workload  completed. 

Source:  GAO  analysis  of  Census  Bureau  data. 

Moreover,  only  a  small  percentage  of  most  local  census  offices’ 
nonresponse  follow-up  workload  was  finished  using  closeout  and  partial 
interviews.  As  shown  in  figure  10,  of  the  499  local  offices  where  reliable 
closeout  data  were  available,^^  413  (83  percent)  reported  that  less  than  2 


^  We  excluded  data  for  those  local  census  offices  that,  according  to  the  bureau,  were  not 
reliable  because  of  various  anomalies,  such  as  inaccurate  coding  of  questionnaires  by  local 
office  staff. 
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percent  of  their  questionnaires  were  finished  in  this  manner,  while  19 
offices  (4  percent)  reported  5  percent  or  more  of  their  finished 
nonresponse  follow-up  work  as  closeout  interviews.  For  partial  interviews, 
of  the  508  offices  where  reliable  data  were  available,  267  (53  percent) 
reported  collecting  less  than  2  percent  of  such  data,  while  47  offices  (9 
percent)  reported  5  percent  or  more  of  their  finished  work  as  partial 
interviews.  The  median  percentages  of  closeout  and  partial  interviews 
were  .8  percent  and  1.9  percent,  respectively. 


Figure  10:  Percentage  of  Local  Census  Offices  Coilecting  Less  Complete  Data 

100  Percentage  of  local  census  offices 


Less  than  2  percent 
2  percent  to  less  than  5  percent 
5  percent  or  more 


Note:  Comparable  data  for  1990  were  not  available  for  comparison  to  2000  results. 
Source:  GAO  analysis  of  Census  Bureau  data. 


At  those  local  census  offices  that  had  substantially  higher  levels  of  closeout 
and  partial  interview  data  than  other  offices,  the  bureau  said  that  some  of 
this  was  understandable  given  the  enumeration  challenges  that  these 
census  offices  faced.  For  example,  according  to  the  bureau,  the  relatively 
high  partial  interview  rate  at  a  New  York  local  office  (3.8  percent  of  that 
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office’s  finished  nonresponse  follow-up  workload)  was  in  line  with  the 
regional  average  of  2.2  percent,  partly  due  to  the  difficulty  that  staff  had  in 
gaining  access  to  apartment  buildings.  Once  building  managers  gave 
enumerators  access  and  they  were  able  to  obtain  information  from  proxies, 
the  number  of  refusals  may  have  decreased,  but  the  number  of  partial 
interviews  increased  because  the  proxies  could  not  provide  complete 
information. 

Still,  as  noted  above,  some  local  census  offices  inappropriately  used  certain 
enumeration  techniques.  For  example,  the  Hialeah,  Florida,  office  reported 
finishing  its  nonresponse  follow-up  workload  in  5  weeks — well  ahead  of 
the  8-week  stretch  goals  and  10  weeks  allotted  for  the  operation.  The 
Homestead,  Florida,  office — where  Hialeah-trained  enumerators  were  later 
transferred  to  help  complete  nonresponse  follow-up — reported  finishing  its 
workload  in  7  weeks.  The  Commerce  Department’s  Office  of  the  Inspector 
General  later  found  that  Hialeah-trained  enumerators  did  not  make  the 
required  number  of  visits  and  telephone  calls  before  contacting  a  proxy  for 
information,  and  did  not  properly  implement  quality  control  procedures 
designed  to  detect  data  falsification.^^  The  bureau  responded  to  these 
findings  by,  among  other  actions,  reworking  over  64,000  questionnaires 
from  the  Hialeah  and  Homestead  offices. 


Questions  Surround 
Whether  Certain 
Reinterview 
Procedures  Were 
Implemented  as 
Intended 


To  help  ensure  that  enumerators  followed  proper  enumeration  procedures 
and  were  not  falsifying  data,  the  bureau  “reinterviewed”  households  under 
certain  circumstances  to  check  enumerators’  work.  As  such,  reinterviews 
were  a  critical  component  of  the  bureau’s  quality  assurance  program  for 
nonresponse  follow-up.  If  falsification  was  detected  during  a  reinterview, 
the  local  office  was  to  terminate  the  enumerator  and  redo  all  of  the 
enumerator’s  work.  Enumerators  making  inadvertent  errors  were  to 
correct  their  mistakes  and  be  retrained.  The  bureau  conducted  three  types 
of  reinterviews: 

I.  Random  reinterviews  were  to  be  performed  on  a  sample  of 

enumerators’  work  during  the  early  weeks  of  their  employment.  Seven 
randomly  selected  questionnaires  from  each  enumerator’s  first  70  cases 
were  to  have  been  reinterviewed. 


For  more  information  on  this  incident,  see  U.S.  Department  of  Commerce,  Office  of 
Inspector  General,  Bureau  of  the  Census:  Re-enumeration  at  Three  Local  Census  Offices 
in  Florida:  Hialeah,  Broward  South,  and  Homestead  CESD-13215-0-0001,  Sept.  29,  2000). 


Page  29 


GAO-02-196  2000  Census  Best  Practices  and  Lessons  Learned 


2.  Administrative  reinterviews  checked  the  work  of  enumerators  whose 
performance  in  certain  dimensions  (e.g.,  the  number  of  partial 
interviews  conducted)  differed  significantly  from  that  of  other 
enumerators  employed  in  the  same  area — and  there  was  no 
justification  for  the  difference.  In  such  cases,  enumerators  could  be 
fabricating  data.  According  to  the  bureau,  administrative  tests  were 
designed  to  identify  enumerators  who  were  making  errors  that  were 
more  likely  to  occur  toward  the  end  of  the  operation,  after  the  random 
check  of  enumerators’  initial  work.  They  were  conducted  at  the 
discretion  of  local  census  officials. 

3.  Supplemental  reinterviews  were  to  be  conducted  at  the  discretion  of 
local  census  officials  when  they  had  some  basis  for  concern  about  the 
quality  of  an  enumerator’s  work. 

On  the  basis  of  our  work  and  that  of  the  bureau,  we  found  that  local  census 
office  officials  often  used  their  discretion  to  not  conduct  administrative 
and  supplemental  reinterviews  and  thus,  a  number  of  local  offices  did  not 
conduct  such  reinterviews.  At  those  offices,  once  the  random  check  of 
enumerators’  initial  work  was  completed,  there  were  no  additional  checks 
specifically  designed  to  catch  enumerators  suspected  of  falsifying  data. 
This  raises  questions  about  the  reinterview  program’s  ability  to  ensure  the 
quality  of  enumerators’  work  over  the  full  duration  of  their  employment  on 
nonresponse  follow-up. 


Local  Managers  Often 
Decided  Against  Conducting 
Administrative  Reinterviews 


Of  the  520  local  census  offices,  52  offices  (10  percent)  conducted  no 
administrative  and  no  supplemental  reinterviews,  according  to  bureau 
data.^®  An  additional  14  offices  (3  percent)  conducted  no  administrative 
reinterviews,  and  an  additional  231  offices  (44  percent)  conducted  no 
supplemental  reinterviews. 


A  chief  in  the  bureau’s  Quality  Assurance  Office  expressed  concern  about 
the  adequacy  of  quality  assurance  coverage  toward  the  end  of  nonresponse 
follow-up  for  offices  that  did  not  conduct  administrative  and  supplemental 
reinterviews.  According  to  this  official,  this  meant  that  once  random 
reinterviews  were  completed  at  those  offices,  there  were  no  additional 
checks  specifically  designed  to  detect  fabricated  data.  Although 


^  In  addition  to  the  511  local  census  offices  located  in  the  United  States,  there  were  9  offices 
in  Puerto  Rico. 
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enumerators’  immediate  supervisors  were  to  check  enumerators’  work 
daily,  these  reviews  were  generally  designed  to  identify  enumerators  who 
were  completing  questionnaires  incorrectly  (e.g.,  not  following  the  proper 
question  sequence  and  writing  illegibly),  whereas  administrative  and 
supplemental  reinterviews  were  aimed  at  identifying  enumerators  who 
were  intentionally  falsifying  data. 

Bureau  officials  said  that  at  those  local  census  offices  that  did  not  conduct 
any  administrative  reinterviews,  local  census  office  managers  could 
conduct  supplemental  reinterviews  if  warranted.  However,  managers 
employed  this  option  infrequently.  Of  the  66  local  offices  that  did  not 
conduct  any  administrative  reinterviews,  just  14  conducted  supplemental 
reinterviews. 

Reasons  that  local  census  managers  could  use — as  specified  by  the 
bureau — for  not  conducting  an  administrative  reinterview  included  (1)  the 
enumerator  no  longer  worked  in  the  area  for  which  the  administrative  test 
was  conducted;  (2)  the  enumerator’s  work  was  characteristic  with  the  area 
(e.g.,  the  enumerator  reported  a  large  number  of  vacant  housing  units  and 
the  area  had  a  large  number  of  seasonal  housing  units);  or  (3)  other  reason, 
with  an  accompanying  explanation.  Managers  were  to  document  their 
decision  on  the  bureau’s  administrative  reinterview  trouble  reports  listing 
the  suspect  enumerators. 

Our  analysis  of  a  week’s  worth  of  administrative  reinterview  trouble 
reports  at  31  local  census  offices  found  that  while  a  number  of  enumerators 
were  flagged  for  administrative  reinterviews,  local  census  office  officials 
typically  decided  against  conducting  them.  Specifically,  of  the  3,784 
enumerators  identified  for  possible  reinterview,  local  officials  subjected 
the  work  of  154  enumerators  (4  percent)  to  reinterviews,  and  passed  on 
3,392  enumerators  (90  percent).  For  306  of  the  3,874  enumerators  (8 
percent)  listed  on  the  administrative  trouble  reports  we  reviewed,  there 
was  no  indication  of  a  final  decision  on  whether  or  not  to  subject  the  future 
work  of  these  enumerators  to  administrative  reinterview. 

Overall,  local  census  offices  conducted  far  fewer  administrative 
reinterviews  than  the  bureau  had  anticipated.  Local  census  offices 
conducted  276,832  administrative  reinterviews — 146,993  (35  percent) 
fewer  than  the  423,825  administrative  reinterviews  the  bureau  had 
expected  based  on  a  number  of  factors,  including  the  number  of  cases 
completed  per  hour  during  the  1990  Census,  and  the  estimated  workload  in 
2000.  Whether  this  was  due  to  better  quality  work  on  the  part  of 
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enumerators,  or  local  managers  deciding  against  subjecting  enumerators’ 
work  to  reinterviews,  is  unknown.  However,  as  administrative 
reinterviews  were  designed  to  detect  fabrication  and  other  quality 
problems  more  likely  to  occur  toward  the  end  of  nonresponse  follow-up 
after  the  random  check  of  enumerators’  initial  work,  it  will  be  important  for 
the  bureau  to  examine  whether  local  census  offices  properly  conducted 
administrative  reinterviews,  and  thus  ensure  the  quality  of  nonresponse 
follow-up  data  throughout  the  duration  of  the  operation. 


Conclusions  Although  nonresponse  follow-up  was  fraught  with  extraordinary 

managerial  and  logistical  challenges,  the  bureau  generally  completed 
nonresponse  follow-up  consistent  with  its  operational  plan — a  remarkable 
accomplishment  given  the  scope  and  complexity  of  the  effort.  Our  review 
highlighted  several  strategies  that  were  key  to  the  bureau’s  success 
including  (1)  an  aggressive  outreach  and  promotion  campaign  and  other 
efforts  aimed  at  boosting  the  mail  response  rate  and  lowering  the  bureau’s 
nonresponse  follow-up  workload;  (2)  a  flexible  recruiting  strategy  that 
made  the  bureau  a  competitive  employer  in  a  tight  national  labor  market; 
(3)  advance  planning  for  addressing  location-specific  enumeration 
challenges;  and  (4)  ambitious  stretch  goals  that  encouraged  local  managers 
to  accelerate  the  pace  of  the  operation.  It  will  be  important  for  the  bureau 
to  document  the  lessons  learned  from  these  initiatives  and  use  them  to  help 
inform  planning  efforts  for  the  next  decennial  census  in  2010. 

It  will  also  be  important  for  the  bureau  to  address  the  continuing  significant 
challenges  that  were  revealed  by  the  conduct  of  nonresponse  follow-up  in 
2000,  including 

•  achieving  an  acceptable  response  rate  (and  thus  lowering  the  bureau’s 
follow-up  workload)  while  controlling  costs; 

•  reversing  the  downward  trend  in  public  participation  in  the  census,  in 
part  by  converting  the  relatively  large  number  of  people  who  are  aware 
of  the  census  into  census  respondents; 

•  keeping  the  address  list  and  maps  used  for  nonresponse  follow-up 
accurate  and  up-to-date; 

•  finding  the  right  mix  of  incentives  to  motivate  local  census  offices  to 
complete  nonresponse  follow-up  on  schedule  without  compromising 
data  quality;  and 

•  ensuring  that  reinterview  procedures  provide  sufficient  quality 
assurance  coverage  through  the  full  duration  of  enumerators’ 
employment  on  nonresponse  follow-up. 
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Recommendations  for 
Executive  Action 


As  the  bureau  plans  for  the  next  national  head  count  in  2010,  we 
recommend  that  the  Secretary  of  Commerce  ensure  that  the  bureau  take 
the  following  actions  to  help  ensure  that  nonresponse  follow-up  is 
conducted  as  cost  effectively  as  possible: 


•  Identify  and  refine  lessons  learned  from  the  2000  nonresponse  follow-up 
operation  and  apply  them  to  the  bureau’s  plans  for  the  2010  Census. 

•  Assess  to  the  extent  practicable,  why  people  who  were  aware  of  the 
census  did  not  return  their  census  questionnaires  and  develop 
appropriate  marketing  countermeasures  to  bridge  the  gap  between  their 
awareness  of  the  census  on  the  one  hand,  and  their  motivation  to 
respond  on  the  other. 

•  Develop  and  test  procedural  and  technological  options  that  have  the 
potential  to  generate  a  more  accurate  and  up-to-date  address  list  and  set 
of  maps  for  nonresponse  follow-up.  As  part  of  this  effort,  the  bureau 
should  explore  how  to  refresh  the  nonresponse  follow-up  address  list 
more  frequently,  even  as  nonresponse  follow-up  is  underway,  so  that 
enumerators  would  not  have  to  make  costly  visits  to  late-responding 
households.  The  bureau  also  needs  to  examine  the  methods  it  uses  in 
activities  that  precede  nonresponse  follow-up  to  develop  and  update  the 
nonresponse  address  list  and  associated  maps.  Specifically,  the  bureau 
should  determine  the  extent  to  which  updates  that  should  have  been 
made  were  properly  reflected  in  the  nonresponse  follow-up  list  and 
maps,  and  take  appropriate  corrective  actions  to  address  any  problems 
it  identifies. 

•  Ensure  that  the  bureau’s  procedures  and  incentives  for  the  timely 
completion  of  nonresponse  follow-up  emphasize  the  collection  of 
quality  data  and  proper  enumeration  techniques  as  much  as  speed. 

•  Examine  the  bureau’s  reinterview  procedures — particularly  as  they 
relate  to  the  discretion  given  to  local  census  officials — ^to  help  ensure 
that  the  procedures  are  sufficient  for  consistently  and  reliably  detecting 
potential  problems  throughout  the  duration  of  enumerators’ 
employment  on  nonresponse  follow-up. 


Agency  Comments  and 
Our  Evaluation 


The  Secretary  of  Commerce  forwarded  written  comments  from  the  Bureau 
of  the  Census  on  a  draft  of  this  report.  The  bureau  concurred  with  all  five 
of  our  recommendations  and  had  no  specific  comments  on  them.  The 
bureau  also  clarified  several  key  points  and  provided  additional 
information  and  perspective,  which  we  incorporated  in  our  report  as 
appropriate. 
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The  bureau  noted  that,  in  addition  to  the  locked  apartment  buildings  that 
we  cited  in  the  Results  in  Brief  section  of  our  report,  gated  communities 
were  also  an  enumeration  challenge.  While  the  body  of  the  report  already 
contained  this  information,  we  added  it  to  the  Results  in  Brief  section  as 
well. 

Our  draft  report  stated:  “One  reason  for  the  errors  in  the  nonresponse 
follow-up  address  lists  was  that  the  bureau  found  it  was  infeasible  to 
remove  late-responding  households.  As  a  result,  enumerators  needed  to 
visit  over  773,000  households  that  had  already  mailed  back  their 
questionnaires. ...”  The  bureau  commented  that  it  made  a  conscious 
decision  to  conduct  these  visits  based  on  logistical  concerns  and,  as  a 
result,  the  bureau  believes  that  our  use  of  the  terms  “errors”  and 
“needlessly”  do  not  take  this  into  consideration  and  are  misleading. 

Because  the  bureau  could  not  refresh  its  nonresponse  follow-up  address 
list  to  reflect  households  that  responded  after  April  18,  the  bureau  had  no 
choice  but  to  send  enumerators  to  those  households  and  collect  the 
information  in-person.  However,  the  term  “needed  to”  better  characterizes 
the  bureau’s  lack  of  options  and  we  revised  the  text  accordingly.  We  also 
deleted  the  term  “errors.” 

In  response  to  our  finding  that  52  local  census  offices  did  not  conduct  any 
reinterviews  after  an  initial  random  check  of  enumerators’  work,  the 
bureau  commented  that  the  initial  random  check  was  not  a  minimal  activity 
in  that  it  involved  reinterviewing  up  to  seven  cases  per  enumerator.  The 
bureau  also  noted  that  there  were  no  operational  requirements  to  conduct 
a  specific  number  of  administrative  or  supplemental  reinterviews.  We 
agree  with  the  bureau’s  comments.  Indeed,  the  draft  report  already 
included  information  on  the  number  of  initial  random  reinterviews  the 
bureau  conducted  and  the  discretionary  nature  of  administrative  and 
supplemental  reinterviews.  Nevertheless,  it  is  also  true,  as  we  note  in  our 
report,  that  once  those  52  local  census  offices  completed  the  seven  random 
reinterviews,  there  were  no  additional  checks  specifically  designed  to 
catch  enumerators  suspected  of  falsifying  data.  Moreover,  we  reported 
that  nationwide,  local  census  offices  conducted  far  fewer  administrative 
reinterviews  than  the  bureau  had  expected.  As  we  note  in  the  report, 
whether  this  was  due  to  the  quality  of  enumerators’  work  or  local  managers 
using  their  discretion  and  opting  not  to  subject  enumerators’  work  to 
reinterviews,  is  unknown. 
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With  respect  to  the  bureau’s  monitoring  of  local  census  office’s 
productivity,  the  bureau  noted  that  headquarters  officials  did  not  work 
directly  with  local  census  office  staff  as  noted  in  the  draft;  rather, 
headquarters  personnel  worked  with  the  bureau’s  regional  census  centers, 
and  they  in  turn  worked  with  the  local  offices.  We  revised  the  text  to 
reflect  this  information. 

With  respect  to  our  observation  that  several  local  census  offices  had  to 
quickly  respond  to  unanticipated  challenges,  such  as  working  with 
nonresponse  follow-up  address  lists  and  maps  that  were  not  accurate  or 
current,  the  bureau  commented  that  there  were  standard  procedures  in  the 
nonresponse  follow-up  enumerator  manual  on  how  to  deal  with 
map/register  discrepancies.  We  verified  this  and  revised  the  text 
accordingly. 

In  describing  the  steps  that  local  census  officials  took  to  encourage  public 
participation  in  the  census,  we  noted  that  census  officials  in  Cleveland  and 
Cincinnati  said  they  provided  additional  training  for  enumerators  on  how 
to  handle  refusals.  The  bureau  noted  that  standardized  training  was 
provided,  across  the  nation,  on  options  for  handling  refusals,  and 
information  was  also  provided  in  the  nonresponse  follow-up  enumerator 
manual.  We  verified  this  information  and  added  it  to  the  report. 

The  bureau  commented  that  the  address  list  and  map  difficulties  that 
enumerators  encountered  were  not  nonresponse  problems  because,  as  we 
note  in  the  report,  and  the  bureau  agrees,  they  should  have  been  dealt  with 
in  earlier  census  operations.  Nevertheless,  the  problems  did  not  surface 
until  nonresponse  follow-up  when  enumerators  encountered  duplicate  and 
nonexistent  addresses,  and  were  less  productive  as  a  result.  For  this 
reason,  the  report  recommends  that  the  bureau  examine  the  methods  it 
uses  in  activities  that  precede  nonresponse  follow-up  to  ensure  the  address 
lists  and  maps  used  for  nonresponse  follow-up  are  accurate  and  up-to-date. 

In  response  to  our  statement  that  nonresponse  follow-up  was  to  help  verify 
changes  to  the  address  list  from  earlier  address  list  development 
operations,  the  bureau  commented  that  nonresponse  follow-up  was 
conducted  to  enumerate  households  from  which  it  did  not  receive  a 
completed  questioimaire;  map  and  address  updates  were  incidental.  We 
agree  with  the  bureau  on  the  primary  purpose  of  nonresponse  follow-up 
and  revised  the  text  to  better  reflect  this  point.  However,  the  bureau’s 
program  master  plan  for  the  master  address  file  includes  nonresponse 
follow-up  as  one  of  a  number  of  address  list  development  and  maintenance 
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operations,  and  the  bureau  expected  enumerators  to  update  maps  and 
address  registers  as  needed  as  part  of  their  field  visits. 

The  bureau  said  it  could  not  confirm  data  in  our  draft  report  on  the  number 
of  vacant  and  deleted  units  identified  during  nonresponse  follow-up  and 
suggested  removing  this  information.  Although  we  obtained  the  data 
directly  from  the  bureau,  given  the  bureau’s  concerns,  we  deleted  the 
section. 

In  commenting  on  the  fact  that  we  did  not  find  a  statistically  significant 
relationship  between  the  week  that  local  census  offices  finished  their 
follow-up  workload  and  the  amount  of  their  residual  workload,  the  bureau 
stated  that  the  report  needed  to  reflect  the  fact  that  residual  nonresponse 
consisted  of  housing  units  for  which  completed  questionnaires  had  not 
been  processed  through  data  capture.  We  revised  the  draft  accordingly. 

The  bureau  noted  that  assistant  managers  for  field  operations,  among  other 
local  census  officials,  could  request  supplemental  reinterviews,  and  not 
just  field  operations  supervisors  as  we  stated  in  our  report.  We  revised  our 
draft  to  include  this  information. 

With  respect  to  our  findings  concerning  the  reinterview  program’s  ability  to 
detect  problems,  particularly  at  the  end  of  nonresponse  follow-up,  the 
bureau  commented  that  there  was  turnover  in  the  enumerator  workforce; 
consequently,  with  new  hires,  random  reinterviews  were  conducted  during 
all  stages  of  the  operation.  As  we  note  in  the  report,  52  local  census  offices 
(about  10  percent  of  all  local  offices),  did  not  conduct  any  administrative 
and  supplemental  reinterviews.  Thus,  once  these  offices  completed  the 
random  reinterviews  on  the  initial  work  of  newly  hired  enumerators,  there 
were  no  additional  checks  specifically  designed  to  catch  enumerators 
suspected  of  falsifying  data.  We  added  language  to  better  clarify  this  point. 

The  bureau  said  that  it  was  uncertain  as  to  the  methodology  and 
documentation  used  for  deriving  figures  on  the  number  of  reinterviews  the 
bureau  conducted.  We  obtained  the  data  from  the  bureau’s  cost  and 
progress  system. 

The  bureau  stated  that  there  was  no  evidence  that  data  quality  was 
compromised  to  motivate  on-time  completion  of  nonresponse  follow-up. 
Our  research  suggests  that  the  impact  of  the  bureau’s  incentives  to 
motivate  timeliness  was  less  clear-cut  given  the  fact  that,  as  we  note  in  our 
report,  (1)  about  40  percent  of  the  local  census  office  managers  believed 


Page  36 


GAO-02-196  2000  Census  Best  Practices  and  Lessons  Learned 


that  scheduling  pressures  had  a  negative  or  significantly  negative  impact  on 
the  quality  of  nonresponse  follow-up,  and  (2)  a  small  number  of  local 
census  offices  took  short-cuts  to  complete  their  work  (which  the  bureau 
later  took  steps  to  rectify).  Thus,  while  we  agree  with  the  bureau  that 
maintaining  data  quality  should  be  a  given  in  determining  motivational 
elements,  the  extent  to  which  the  bureau  accomplished  this  goal  for 
nonresponse  follow-up  appeared  to  have  had  mixed  results. 

In  commenting  on  our  conclusion  that  it  will  be  important  for  the  bureau  to 
ensure  that  reinterview  procedures  provide  sufficient  quality  assurance 
through  the  full  duration  of  nonresponse  follow-up,  the  bureau  noted  that 
the  reinterview  operation  must  be  designed  to  provide  sufficient  quality 
assurance  coverage.  We  revised  the  text  accordingly 


We  are  sending  copies  of  this  report  to  the  Honorable  Dan  Miller  and 
Carolyn  B.  Maloney,  House  of  Representatives,  and  those  in  other 
interested  congressional  committees;  the  Secretary  of  Commerce;  and  the 
Acting  Director  of  the  Bureau  of  the  Census.  Copies  will  be  made  available 
to  others  on  request.  Major  contributors  to  this  report  are  included  in 
appendix  III.  If  you  have  any  questions  concerning  this  report,  please  call 
me  on  (202)  512-6806. 

/! 

Patricia  A.  Dalton 
Director 
Strategic  Issues 
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Appendix  I 

Local  Census  Offices  Included  in  This  Review 


Local  Census  Offices  in  the 
Census  Bureau’s  Atlanta 
Region 


Atlanta  East 
Bradenton 
Fort  Myers 


Local  Census  Offices  in  the 
Census  Bureau’s  Boston 
Region 


Boston  North 
Burlington 
Hartford 
Providence 


Local  Census  Offices  in  the 
Census  Bureau’s  Charlotte 
Region 


Ashland-Hanover 

Beaufort 

Conway 

Greenville,  North  Carolina,  East 
Greenville,  North  Carolina,  West 
Wilmington 


Local  Census  Offices  in  the 
Census  Bureau’s  Chicago 
Region 


Chicago  Central 

Chicago  Far  North 

Chicago  Near  North 

Chicago  Near  South 

Chicago  Near  Southwest 

Chicago  West 

Indianapolis 

Midland 

Milwaukee 

Superior 


Local  Census  Offices  in  the 
Census  Bureau’s  Dallas 
Region 


Corpus  Christi 
Dallas  Central 
Greenville,  Mississippi 
Harris  County,  Northeast 
Laredo 
McAllen 

New  Orleans  Central 
Orleans  Parish 
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Appendix  I 

Local  Census  Offices  Included  in  This  Review 


Local  Census  Offices  in  the 
Census  Bureau’s  Denver 
Region 


Flagstaff 
Las  Cruces 
Las  Vegas 
Phoenix  South 
Santa  Fe 
Yuma 


Local  Census  Offices  in  the 
Census  Bureau’s  Detroit 
Region 


Cincinnati 

Cleveland 

Marquette 


Local  Census  Offices  in  the 
Census  Bureau’s  Kansas 
City  Region 


Kansas  City 
Moorhead 
St.  Louis  City 


Local  Census  Offices  in  the 
Census  Bureau’s 
Philadelphia  Region 


Baltimore  West 
Philadelphia  North 
Philadelphia  South 
Pittsburgh 


Local  Census  Offices  in  the 
Census  Bureau’s  Los 
Angeles  Region 


Hollywood/Mid- Wilshire 
Los  Angeles  Downtown 
Santa  Monica 


Local  Census  Offices  in  the 
Census  Bureau’s  New  York 
Region 


Bronx  Northeast 
Brooklyn  Central 
Brooklyn  East 
Brooklyn  Northeast 
New  York  East 
New  York  North 
New  York  Northeast 
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Local  Census  Offices  in  the 
Census  Bureau’s  Seattle 
Region 


Portland 

San  Francisco  Northeast 
San  Francisco  Southeast 
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Comments  from  the  U.S.  Department  of  Commerce 
U.S.  Census  Bureau 


U.S.  General  Accounting  Office  draft  report  entitled  2000  Census:  Best  Practices  and 
Lessons  Learned  for  More  Cost-Effective  Nonresponse  Foilow-up 

Comments  on  the  Text  of  the  Report 

1 .  Section:  Page  3,  Bullet  3  -  [The  Bureau]  “called  on  local  census  offices  to  identify  local 
enumeration  challenges,  such  as  locked  apartment  buildings,  and  to  develop  action  plans 
to  address  them . . . 


Comment:  Gated  communities  also  were  identified  as  an  enumeration  challenge. 

Section:  Page  4,  Paragraph  3,  continued  on  Page  5  -  “Third,  the  address  lists  used  for 
nonresponse  follow-up  did  not  always  contain  the  latest  available  information,  and 
associated  maps  used  by  census  enumerators  during  nonresponse  follow-up  contained 
inaccuracies.  One  reason  for  the  errors  in  the  nonresponse  follow-up  address  lists  was 
that  the  Bureau  found  it  was  infeasible  to  remove  late-responding  households.  As  a 
result,  enumerators  needlessly  visited  over  773,000  households  that  had  already  mailed 
back  their  questionnaires — an  effort  that  approached  $22  million  in  additional  costs  for 
nonresponse  follow-up,  based  on  our  estimate,  and  confused  respondents.” 

Comment:  The  detennination  that  it  was  infeasible  to  remove  late-responding 
households  was  a  conscious  decision  based  on  logistical  concerns.  Use  of  the  temis 
“errors”  and  “needlessly”  do  not  take  this  into  consideration  and  are  misleading. 

Section:  Page  6,  Paragraph  1  -  ‘Tinally,  questions  surround  the  extent  to  which  certain 
reinterview  procedures  were  implemented  throughout  the  entire  nonresponse  follow-up 
operation  as  intended,  as  local  census  office  managers  often  exercised  their  diseretion  and 
opted  against  eonducting  this  key  quality  assurance  procedure  aimed  at  detecting 
enumerator  firaud.  For  example,  52  loeal  census  offices  (about  10  percent  of  all  local 
offices)  did  not  conduct  any  reinterviews  after  an  initial  random  check  of  enumerators’ 
work.  A  senior  Bureau  quality  assurance  official  expressed  concerns  about  the  adequacy 
of  quality  assurance  coverage  toward  the  end  of  nonresponse  follow-up  at  these  offices.” 

Comment:  The  initial  random  eheck  was  not  a  minimal  aetivity.  The  check  involved 
reinterview  of  up  to  seven  cases  per  enumerator.  There  were  no  operational  requirements 
to  conduct  a  specific  number  of  administrative  or  supplemental  reinterviews. 

Section:  Page  22,  Paragraph  1  -  “Once  nonresponse  follow-up  began.  Bureau  officials 
tracked  production  rates  as  the  primary  measure  of  whether  loeal  offices  had  met  their 
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stalling  goals.  For  example,  Bureau  officials  said  that  both  Bureau  headquarters  and 
regional  census  center  staff  monitored  local  census  offices’  production  daily.  If  an  office 
was  not  meeting  its  production  goals,  Bureau  headquarters  officials  worked  with 
managers  of  that  office  to  determine  the  reasons  for  the  shortfall  and  the  actions  necessary 
to  increase  production.” 

Comment:  Census  Bureau  headquarters  and  regional  census  center  (RCC)  staff  did 
monitor  local  census  offices  (LCDs);  however,  headquarters’  officials  worked  with  the 
RCCs,  who  worked  with  the  LCDs.  Headquarters  persoimel  did  not  work  directly  with 
LCO  staff. 

5.  Section:  Page  23,  Paragraph  2  -  “A  third  factor  that  contributed  to  the  timely  completion 
of  nonresponse  follow-up  was  preparing  in  advance  for  probable  enumeration  challenges. 
To  do  this,  the  Bureau  called  on  local  census  offices  and  their  respective  regional  census 
centers  to  develop  action  plans  that,  among  other  things,  identified  hard-to-enumerate 
areas  within  their  jurisdictions,  such  as  immigrant  neighborhoods,  and  propose  strategies 
for  dealing  with  those  challenges.  These  strategies  included  such  methods  as  paired/team 
enumeration  for  high-crime  areas,  and  hiring  bilingual  enumerators.  While  this  early 
planning  effort  helped  local  census  offices  reaet  to  anticipated  enumeration  challenges, 
several  local  census  offices  also  had  to  quickly  respond  to  unanticipated  challenges,  such 
as  working  with  nonresponse  follow-up  address  lists  and  maps  that  were  not  accurate  or 
current.” 

Comment:  Most  LCOs  had  to  quickly  meet  unanticipated  challenges;  however,  there 
were  standard  procedures  in  the  nonresponse  follow-up  (NRFU)  enumerator  manual  on 
how  to  deal  with  map/register  discrepancies. 

6.  Section:  Page  25,  Paragraph  1  -  “Local  census  officials  encouraged  public  participation 
with  a  variety  of  approaches.  For  example,  census  officials  in  Cleveland  and  Cincinnati 
said  they  provided  additional  training  for  enumerators  on  how  to  handle  refusals  and 
practiced  what  was  taught  in  mock  interviews.  Officials  in  other  census  offices  said  they 
partnered  with  local  community  leaders  who  subsequently  helped  reach  out  to  hard-to- 
enumerate  groups,  hired  people  who  were  bilingual  or  otherwise  trusted  and  known  by 
residents,  and  held  media  campaigns.  Overall,  according  to  Bureau  data,  close  to 
470,000  households  of  the  approximately  42  million  making  up  the  nonresponse  follow¬ 
up  workload  (about  1  percent),  refused  to  participate  in  the  census.” 

Comment:  Standardized  training  was  provided,  across  the  Nation,  on  options  for 
handling  refusals.  This  information  also  was  provided  in  written  form  in  the  NRFU 
enumerator  manual. 

7.  Section:  Page  26,  Paragraph  2,  continued  on  Page  27  -  “Among  the  more  frequent 
problems  managers  cited  were  duplicate  addresses  and  changes  not  being  made  from 
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prior  operations.  For  example,  at  a  local  census  office  in  the  Seattle  region, 
managers  said  that  some  addresses  were  residences  or  businesses  that  had  been  gone  for 
10-15  years  and  should  have  been  deleted  in  previous  census  operations  but  were  not.” 

Comment:  These  address  list  and  map  “problems”  are  not  really  NRFU  problems.  The 
end  of  the  paragraph  indicates  clearly  that  some  addresses  “. . .  should  have  been  deleted 
in  previous  census  operations.” 

8.  Section:  Page  28,  Paragraph  2,  continued  on  Page  29  -  “The  Bureau  developed  its  master 
address  list  and  maps  using  a  series  of  operations  throughout  the  decade,  each  designed  to 
add  incremental  improvements.  Nonresponse  follow-up  was  to  help  verify  changes  to  the 
address  list  from  some  of  these  earlier  operations.  As  a  result,  the  Bureau  was  expecting 
some  discrepancies  between  the  nonresponse  follow-up  address  list  and  what  enumerators 
found  in  the  field  when  they  went  door-to-door,  which  could  account  for  some  of  the 
local  census  officials’  perceptions.  Of  the  approximately  119  million  questionnaires 
delivered,  3.1  million  were  to  units  subsequently  found  during  nonresponse  follow-up  to 
be  vacant  and  1.9  million  were  deleted  (e.g.,  because  they  were  found  to  be  nonexistent 
units),  according  to  Bureau  data.” 

Comment:  The  NRFU  was  conducted  to  enumerate  households  from  which  we  had  not 
received  a  completed  questionnaire.  It  was  not  conducted  to  “. . .  help  verify  changes  to 
the  address  list.  .  .  .  ”;  map  and  address  updates  were  incidental  and  “problems”  were 
renmants  of  earlier  operations  as  indicated  in  the  previous  item.  Furthennore,  we  cannot 
confirm  the  numbers  cited  for  vacant  and  deleted  units  identified  during  NRFU  in  2000. 
However,  we  can  confinn  that  these  numbers  are  too  low,  based  on  the  fact  that  about 
8  million  vacant  and  deleted  units  were  identified  as  such  for  the  first  time  during  NRFU. 
We  recommend  that  this  paragraph  be  deleted,  given  the  concerns  noted. 

9.  Section:  Page  29,  Paragraphs  2  and  3  -  “Another  factor  that  affected  the  currency  of  the 
nonresponse  follow-up  address  list  was  the  cut-off  date  for  mail-back  responses.  The 
Bureau  set  April  1 1,  2000,  as  the  deadline  for  mail-back  responses  for  purposes  of 
generating  the  address  list  for  nonresponse  follow-up.  However,  according  to  Bureau 
officials,  the  Bureau  got  an  unexpected  boost  from  its  outreach  and  promotion  campaign, 
which  stressed  the  importance  of  cooperating  with  census  enumerators.  As  a  result,  by 
April  30-almost  2  weeks  after  the  April  18  printing  of  the  nonresponse  follow-up  address 
list  for  late  mail  retums-the  Bureau  had  received  an  additional  773,784  questionnaires. 
Bureau  headquarters  officials  told  us  it  was  not  feasible  to  remove  these  from  the  address 
lists  and  thus,  enumerators  visited  these  households.  The  cost  to  the  Bureau  of  these 
otherwise  needless  visits  approached  $22  million  . .  .  .” 

Comment:  Some  addresses  of  late  returns  were  removed  from  the  D-166  report  form; 
however,  even  after  the  new  report  was  generated,  we  continued  to  receive  late  mail 
returns.  As  mentioned  earher,  the  reference  to  “needless  visits”  implies  arbitrary  inaction 


Page  44 


GAO-02-196  2000  Census  Best  Practices  and  Lessons  Learned 


Appendix  II 

Comments  from  the  Seeretary  of  Commerce 


4 

in  allowing  enumerators  to  visit  these  households  as  opposed  to  a  conscious  decision 
based  on  logistical  concerns, 

10.  Section:  Page  34,  (full)  Paragraph  2  -  “Neither  did  we  find  a  statistically  significant 
relationship  between  the  week  that  local  census  offices  finished  their  nomesponse  follow¬ 
up  workload  and  the  amount  of  residual  workload  [footnote],  they  had,  if  any.” 

Comment:  The  second  sentence  needs  to  reflect  the  fact  that  residual  nonresponse 
consisted  of  units  for  which  completed  questionnaires  had  not  been  processed  through 
data  capture. 

1 1 .  Section:  Page  39,  Item  (3)  -  “Supplemental  reinterviews  were  to  be  conducted-also  at 
local  census  managers’  discretion-when  local  census  personnel,  called  field  operations 
supervisors,  requested  that  such  reinterviews  be  done  because  they  had  some  basis  for 
concern  about  the  quality  of  an  enumerator’s  work.” 

Comment:  Quality  reinterviews  could  be  requested  by  assistant  managers  for  field 
operations  or  others  in  addition  to  field  operations  supervisors. 

12.  Section:  Page  39,  Last  Paragraph  -  “On  the  basis  of  our  work  and  that  of  the  Bureau,  we 
found  that  local  census  office  managers  often  used  their  discretion  to  not  conduct 
administrative  and  supplemental  reinterviews.  As  a  result,  a  number  of  offices  did  not 
conduct  any  administrative  or  supplemental  reinterviews,  which  raises  questions  about 
the  ability  of  the  reinterview  program  to  detect  problems,  particularly  towards  the  end  of 
the  nonresponse  follow-up  operation.” 

Comment:  A  statement  needs  to  be  added  indicating  that  there  was  turnover  in  the 
enumerator  workforce;  hence,  with  new  hires,  random  reinterview  was  conducted  during 
all  stages  of  the  operation. 

13.  Section:  Page  42,  Paragraph  1  -  “Overall,  local  census  offices  conducted  far  fewer 
administrative  reinterviews  than  the  Bureau  had  anticipated.  Local  census  offices 
conducted  276,832  administrative  reinterviews-l  46,993  (35  percent)  fewer  than  the 
423,825  administrative  reinterviews  the  Bureau  had  expected  based  on  a  number  of 
factors,  including  the  number  of  cases  completed  per  hour  during  the  1990  Census,  and 
the  estimated  workload  in  2000.  Whether  this  was  due  to  better  quality  work  on  the  part 
of  enumerators,  or  local  managers  deciding  against  subjecting  enumerators’  work  to 
reinterviews,  is  unknown.  However,  as  administrative  reinterviews  were  designed  to 
detect  problems  more  likely  to  occur  at  the  end  of  nonresponse  follow-up,  it  will  be 
important  for  the  Bureau  to  examine  whether  local  census  offices  properly  conducted 
administrative  reinterviews,  and  thus  ensured  the  quality  of  nomesponse  follow-up  data 
throughout  the  duration  of  the  operation.” 
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Comment:  We  are  uncertain  as  to  the  methodology  for  deriving  this  estimate  and  the 
documentation  from  which  it  was  obtained. 

14.  Section:  Page  43,  Bullet  4  -  “[It  will  also  be  important  for  the  Bureau  to  address  .... 
including]  finding  the  right  mix  of  ineentives  to  motivate  local  census  offices  to  complete 
nonresponse  follow-up  on  schedule  without  compromising  data  quality. . . .” 

Comment:  There  is  no  evidence  that  data  quality  was  compromised  to  motivate  on-time 
completion  of  NRFU.  Not  compromising  the  quality  of  data  should  be  a  given  in 
determining  motivational  elements. 

15.  Section:  Page  43,  Bullet  5  -  “[It  will  also  be  important  for  the  Bureau  to  address  .  . . , 
including]  ensuring  that  reinterview  procedures  provide  sufficient  quality  assurance 
through  the  full  duration  of  nonresponse  follow-up.” 

Comment:  The  reinterview  operation  must  be  designed  to  provide  sufficient  quality 
assurance  coverage. 

Responses  to  GAO  Recommendations 

Census  Bureau  Response:  The  Census  Bureau  concurs  with  the  recommendations  and  has  no 
specific  comments  on  them. 
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